Comments on Kreiner 2011: Is the foundation under PISA solid? A critical look at the scaling model underlying international comparisons of student attainment

i This paper is concerned with two issues:  Do the outcomes of PISA – more particularly the rankings – depend on the items chosen?  Could alternative scaling procedures, produce different outcomes – more particularly the rankings. A careful examination of these two issues is clearly an important activity that should be the focus of rigorous scholarly investigation. Unfortunately this article presents neither a rigorous nor a scholarly investigation. In making comments on the article I shall organise them into two sections. First, I shall make some general points about the work that cover the implications for PISA. Second, I shall make some more specific points that relate to the paper's acceptability as an academic piece of work. General Comments The fundamental flaw in Kreiner's argument is that he confounds the two primary issues of: (a) Do the outcomes of PISA depend upon the set of items that are developed and chosen?; and (b) Does the use of the Rasch model provide misleading results because the data do not fit the Rasch model? The PISA Consortium has undertaken considerable work on both of these issues and the evidence demonstrates that while the items matter enormously, the model matters to a much lesser degree. In what ways does item choice matter? One of the goals of PISA is to compare the outcomes of education systems. A fundamental issue, therefore, is what should form the basis of that comparison. PISA has taken the position that the basis of the comparison should be a relatively large basket of items that are representative of the knowledge, skills and understandings that reflect the respective content framework and are valued by the set of countries that participate as well as leading educators. The procedures for preparing this basket of items have been fully articulated in Chapter 2 of each of the PISA technical reports and collections of sample items have been regularly published, for example OECD, 2009a, 2009b.